Papers with Romanian distilled
Distilling the Knowledge of Romanian BERTs Using Multiple Teachers (2022.lrec-1)
Copied to clipboard
Andrei-Marius Avram, Darius Catrina, Dumitru-Clementin Cercel, Mihai Dascalu, Traian Rebedea, Vasile Pais, Dan Tufis
| Challenge: | Existing approaches to train pre-trained language models focus on the English language, thus widening the gap when considering low-resource languages. |
| Approach: | They propose three versions of distilled BERT models for the Romanian language . they argue that the models offer performance comparable to their teachers . |
| Outcome: | The proposed models perform comparable to their teachers, while being twice as fast on a GPU and 35% smaller. |